2023-12-08 JSON-RPC endpoint unavailable

Incident summary

From Fri Dec 8th 3:40pm CET to Sat Dec 9th 12:40PM our services were unable to make successful requests to GLIF’s JSON-RPC endpoint, blocking any interaction with the central smart contract. The GLIF team was quickly made aware of the issue, but due to higher-than-usual chain activity it took a bit to recover. At the time, no other known public JSON-RPC endpoint worked as temporary replacement.

Impact

There was only one Impact Evaluator round during the incident, leading to lower-than-usual rewards for Station node operators.

Corrective actions

• Automatic JSON-RPC endpoint fallback was put in place, falling back to https://chain.love/ should GLIF’s endpoint be unavailable

• More endpoints will be added, should both configured endpoints become unavailable

• Running our own JSON-RPC node was considered, but for the time dismissed, because of the operational overhead